[1] 0.9772499
Hypothesis Testing
Tao Lin
Office Hours: Fri 1:30 - 3:30 PM Smith 35
Section Slides URL: soxv/CSSS-321-Labs
Agenda
Hypothesis Testing
T = \frac{\hat{\theta} - \theta_0}{\hat{\text{se}}(\hat{\theta})}
p\text{-value} = \text{Pr}(|T| > z_{\alpha/2})
| Reject H_0 | Retain H_0 | |
|---|---|---|
| H_0 is True | False Positive/Type I Error (\alpha) | True Negative |
| H_0 is False | True Positive | False Negative/Type II Error (\beta) |
A malicious researcher in a big platform company first test for an overall effect. Had the null been rejected, the researcher would report the finding. If the researcher fails to reject, then he run a test for a subgroup and report that result.
Assuming the two tests are independent, we have
\begin{aligned} \text{Pr}(\text{at least 1 reject}) =& 1 - \text{Pr}(\text{no reject}) \\ =& 1 - (1 - \alpha)^2 \\ =& 1 - 0.95 \times 0.95 \\ =& 0.0975 > 0.05 \end{aligned}
The Type 1 error goes up because of “hidden” multiple testing. The researcher can’t commit to the same data analysis plan under di↵erent realizations of the sample.
broockman_kalla_2016_w9.csv.t.test() to perform two-sample two-sided t-test for wave 0 (placebo test), wave 1, wave 2, wave3, and wave 4.power.t.test() and set the number of observation per group to 900 (n = 900), set the standard deviation to 30 (sd = 30). How can we interpret the result?geom_pointrange() in ggplot2 to plot the point estimate and confidence interval for each wave.therm_trans_t* ~ treat_ind + therm_trans_t0 + vf_age + vf_racename + vf_female + vf_democrat
treat_ind. To extract point estimate of coefficent, we need to use coef(), and to extract the standard error of coefficient, we need to use the following code:sqrt(diag(vcov(model)))["treat_ind"]
treat_ind in regression models. What do you find?library(tidyverse)
dat <- read.csv("./data/broockman_kalla_2016_w9.csv")
t0 <- t.test(dat$therm_trans_t0[dat$treat_ind==1], dat$therm_trans_t0[dat$treat_ind==0])
t1 <- t.test(dat$therm_trans_t1[dat$treat_ind==1], dat$therm_trans_t1[dat$treat_ind==0])
t2 <- t.test(dat$therm_trans_t2[dat$treat_ind==1], dat$therm_trans_t2[dat$treat_ind==0])
t3 <- t.test(dat$therm_trans_t3[dat$treat_ind==1], dat$therm_trans_t3[dat$treat_ind==0])
t4 <- t.test(dat$therm_trans_t4[dat$treat_ind==1], dat$therm_trans_t4[dat$treat_ind==0])
power.t.test(n = 900, delta = t0$estimate[2] - t0$estimate[1], sd = 30)
Two-sample t test power calculation
n = 900
delta = 0.6644184
sd = 30
sig.level = 0.05
power = 0.06805953
alternative = two.sided
NOTE: n is number in *each* group
Two-sample t test power calculation
n = 900
delta = 6.75578
sd = 30
sig.level = 0.05
power = 0.9975577
alternative = two.sided
NOTE: n is number in *each* group
t_ci_dat <- data.frame(
wave = c(0, 1, 2, 3, 4),
est = c(
t0$estimate[1] - t0$estimate[2],
t1$estimate[1] - t1$estimate[2],
t2$estimate[1] - t2$estimate[2],
t3$estimate[1] - t3$estimate[2],
t4$estimate[1] - t4$estimate[2]
),
lwr = c(t0$conf.int[1], t1$conf.int[1], t2$conf.int[1], t3$conf.int[1], t4$conf.int[1]),
upr = c(t0$conf.int[2], t1$conf.int[2], t2$conf.int[2], t3$conf.int[2], t4$conf.int[2])
)
reg_t1 <- lm(therm_trans_t1 ~ treat_ind + therm_trans_t0 + vf_age + vf_racename + vf_female + vf_democrat, data = dat)
reg_t2 <- lm(therm_trans_t2 ~ treat_ind + therm_trans_t0 + vf_age + vf_racename + vf_female + vf_democrat, data = dat)
reg_t3 <- lm(therm_trans_t3 ~ treat_ind + therm_trans_t0 + vf_age + vf_racename + vf_female + vf_democrat, data = dat)
reg_t4 <- lm(therm_trans_t4 ~ treat_ind + therm_trans_t0 + vf_age + vf_racename + vf_female + vf_democrat, data = dat)
reg_ci_dat <- data.frame(
wave = c(0, 1, 2, 3, 4),
est = c(
0,
coef(reg_t1)["treat_ind"],
coef(reg_t2)["treat_ind"],
coef(reg_t3)["treat_ind"],
coef(reg_t4)["treat_ind"]
),
lwr = c(
0,
coef(reg_t1)["treat_ind"] - qnorm(0.975) * sqrt(diag(vcov(reg_t1)))["treat_ind"],
coef(reg_t2)["treat_ind"] - qnorm(0.975) * sqrt(diag(vcov(reg_t2)))["treat_ind"],
coef(reg_t3)["treat_ind"] - qnorm(0.975) * sqrt(diag(vcov(reg_t3)))["treat_ind"],
coef(reg_t4)["treat_ind"] - qnorm(0.975) * sqrt(diag(vcov(reg_t4)))["treat_ind"]
),
upr = c(
0,
coef(reg_t1)["treat_ind"] + qnorm(0.975) * sqrt(diag(vcov(reg_t1)))["treat_ind"],
coef(reg_t2)["treat_ind"] + qnorm(0.975) * sqrt(diag(vcov(reg_t2)))["treat_ind"],
coef(reg_t3)["treat_ind"] + qnorm(0.975) * sqrt(diag(vcov(reg_t3)))["treat_ind"],
coef(reg_t4)["treat_ind"] + qnorm(0.975) * sqrt(diag(vcov(reg_t4)))["treat_ind"]
)
)
ci_dat <- bind_rows(t_ci_dat, reg_ci_dat, .id = "type") %>%
mutate(type = ifelse(type == 1, "t.test", "regression"))
ggplot(aes(x = wave, y = est, ymin = lwr, ymax = upr, group = type), data = ci_dat) +
geom_pointrange(aes(color = type), position = position_dodge(width = 0.2))
CSSS/SOC/STAT 321 Data Science and Statistics for Social Scicence I